Skip to content

feat(benchmark): harden PersonaMem AML evaluation - #37

Merged
Peefy merged 1 commit into
mainfrom
feat/aml-personamem-evaluation
Sep 11, 2026
Merged

feat(benchmark): harden PersonaMem AML evaluation#37
Peefy merged 1 commit into
mainfrom
feat/aml-personamem-evaluation

Conversation

@hrw991009

Copy link
Copy Markdown
Collaborator

Summary

  • add a complete PersonaMem-v2 AML-local MCQ evaluation report
  • make sharded PersonaMem answering honor OPENROUTER_ANSWER_MODEL, with the existing Qwen default as fallback
  • make resumed answer sharding remove only stale generated input shards while preserving all persisted answer artifacts
  • add strict shard-answer merging with duplicate, missing-ID, and unexpected-ID validation before MCQ scoring

Result

  • completed 5,000 / 5,000 questions
  • official MCQ accuracy: 1,945 / 5,000 (38.90%)
  • 60 empty provider answers (1.20%)
  • document that this AML-local result is not leaderboard-comparable and cannot claim Recall/Hit without QA-level source IDs

Validation

  • python -m py_compile benchmark/aml-local/reshard_answers.py benchmark/aml-local/merge_answers.py
  • merged and validated 5,000 unique answer IDs before scoring
  • ran vendored PersonaMem MCQ exact scoring for all 5,000 records
  • git diff --check

@Peefy
Peefy merged commit e3e7576 into main Sep 11, 2026
13 checks passed
@Peefy
Peefy deleted the feat/aml-personamem-evaluation branch September 11, 2026 04:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants